Improving measurement-based care implementation in youth mental health through organizational leadership and climate: a mechanistic analysis within a randomized trial

Background Theory and correlational research indicate organizational leadership and climate are important for successful implementation of evidence-based practices (EBPs) in healthcare settings; however, experimental evidence is lacking. We addressed this gap using data from the WISDOM (Working to Implement and Sustain Digital Outcome Measures) hybrid type III effectiveness-implementation trial. Primary outcomes from WISDOM indicated the Leadership and Organizational Change for Implementation (LOCI) strategy improved fidelity to measurement-based care (MBC) in youth mental health services. In this study, we tested LOCI’s hypothesized mechanisms of change, namely: (1) LOCI will improve implementation and transformational leadership, which in turn will (2) mediate LOCI’s effect on implementation climate, which in turn will (3) mediate LOCI’s effect on MBC fidelity. Methods Twenty-one outpatient mental health clinics serving youth were randomly assigned to LOCI plus MBC training and technical assistance or MBC training and technical assistance only. Clinicians rated their leaders’ implementation leadership, transformational leadership, and clinic implementation climate for MBC at five time points (baseline, 4-, 8-, 12-, and 18-months post-baseline). MBC fidelity was assessed using electronic metadata for youth outpatients who initiated treatment in the 12 months following MBC training. Hypotheses were tested using longitudinal mixed-effects models and multilevel mediation analyses. Results LOCI significantly improved implementation leadership and implementation climate from baseline to follow-up at 4-, 8-, 12-, and 18-month post-baseline (all ps < .01), producing large effects (range of ds = 0.76 to 1.34). LOCI’s effects on transformational leadership were small at 4 months (d = 0.31, p = .019) and nonsignificant thereafter (ps > .05). LOCI’s improvement of clinic implementation climate from baseline to 12 months was mediated by improvement in implementation leadership from baseline to 4 months (proportion mediated [pm] = 0.82, p = .004). Transformational leadership did not mediate LOCI’s effect on implementation climate (p = 0.136). Improvement in clinic implementation climate from baseline to 12 months mediated LOCI’s effect on MBC fidelity during the same period (pm = 0.71, p = .045). Conclusions LOCI improved MBC fidelity in youth mental health services by improving clinic implementation climate, which was itself improved by increased implementation leadership. Fidelity to EBPs in healthcare settings can be improved by developing organizational leaders and strong implementation climates. Trial registration ClinicalTrials.gov identifier: NCT04096274. Registered September 18, 2019. Supplementary Information The online version contains supplementary material available at 10.1186/s13012-024-01356-w.


Background
Many implementation theories and frameworks in healthcare assert the importance of organizational leadership and organizational implementation climate for achieving high fidelity to newly implemented clinical interventions [1][2][3][4][5][6]; however, the evidentiary basis for these claims is thin.The usual standard for making causal claims in the medical and social sciences is demonstration of effect within a randomized controlled trial [7]; yet, recent reviews indicate no trials have tested whether experimentally induced change in organizational leadership or organizational implementation climate contributes to improved implementation of clinical interventions in healthcare [8,9].In the present study, we address this gap using data from the WISDOM (Working to Implement and Sustain Digital Outcome Measures) hybrid type III effectiveness-implementation trial.The WISDOM trial showed that a strategy called Leadership and Organizational Change for Implementation (LOCI) [10,11], which targets organizational leadership and organizational implementation climate, improved fidelity to measurement-based care (MBC) in outpatient mental health clinics serving youth [12].In this paper, we tested LOCI's hypothesized mechanisms of change.Specifically, we tested whether improvement in clinic leadership contributed to improvement in clinic implementation climate and whether improved implementation climate in turn contributed to improved MBC fidelity.

Measurement-based care in youth mental health
Measurement-based care is an evidence-based practice (EBP) that involves the collection of standardized symptom rating scales from patients prior to each treatment session and use of the results to guide treatment decisions [13].Meta-analyses of over 30 randomized controlled trials indicate feedback from MBC improves the outcomes of mental health treatment relative to services as usual across patient ages, diagnoses, and intervention modalities [14][15][16][17].There is also evidence MBC improves mental health medication adherence [18], reduces risk of treatment dropout [14], and is particularly effective for youths and for patients who are most at risk for treatment failure [14][15][16]18].
Unfortunately, MBC is rarely used in practice.Only 14% of clinicians who deliver mental health services to youth in the USA use any form of MBC [19], and MBC usage rates are similarly low in other countries [20,21].When MBC use is mandated, less than half of clinicians view feedback and use it to guide treatment [22][23][24].Digital MBC systems (i.e., measurement feedback systems) remove many practical barriers to MBC implementation by collecting measures from patients electronically (e.g., via tablet or phone) and instantaneously generating feedback [25].However, even when these systems are available, clinician fidelity to MBC-defined as administering measures, viewing feedback reports, and using the information to guide treatment-is often substandard [22,26].Qualitative and quantitative studies of MBC implementation indicate clinicians' work environments explain much of the variation in their attitudes toward, and use of, MBC [19,27,28], with organizational leadership and supportive organizational culture or organizational implementation climate identified as key determinants [13,[28][29][30].

Mechanisms of the LOCI strategy
LOCI is a multicomponent organizational implementation strategy that engages organizational executives and first-level leaders (i.e., those who administratively supervise clinicians) to build an organizational climate to support the implementation of a focal EBP with fidelity [10,31].It includes two overarching components: (1) monthly organizational strategy meetings between executives and LOCI consultants/trainers to develop and embed policies, procedures, and practices that support implementation of a focal EBP and ( 2) training and coaching for first-level leaders, to develop their skills in leading implementation.Organizational survey data and feedback guide planning, goal specification, and progress monitoring for both components.The aim of these components is to develop an organizational implementation climate [32,33] in which clinicians perceive that use of a specific EBP with high fidelity is expected, supported, and rewarded [34].
Figure 1 shows LOCI's theoretical model as applied to MBC in the present study.This model forms the basis for the study hypotheses.The LOCI strategy draws on two leadership theories-full-range leadership [35,36] and implementation leadership [37]-and on theories of organizational implementation climate [33,34], to explain variation in implementation success and to identify targets for implementation improvement.As is shown in Fig. 1, LOCI seeks to equip firstlevel leaders (e.g., clinical program managers), with two types of leadership behaviors believed to influence implementation success.Transformational leadership, drawn from the full-range leadership model, is a general type of leadership that reflects a leader's ability to inspire and motivate employees to follow an ideal or course of action [38,39].Implementation leadership is a type of focused leadership that refers to leader behaviors that facilitate the organization's specific strategic objective of successfully implementing a focal EBP, such as MBC [37,[40][41][42][43][44].As is shown in the figure, LOCI aims to increase first-level leaders' use of these leadership behaviors in order to support the development of an implementation climate that prompts and supports clinicians' use of the focal EBP with high fidelity.
Correlational studies offer preliminary support for the relationships shown in Fig. 1.In a 5-year study of 30 outpatient mental health clinics serving youth, Williams et al. [45] showed that increases in implementation leadership at the clinic level were associated with increases in clinic EBP implementation climate, which subsequently predicted increases in clinicians' selfreported use of evidence-based psychotherapy techniques.Other studies have shown that higher levels of organizational implementation climate predict higher observed fidelity to evidence-based mental health interventions in outpatient clinics and schools [46,47].In the other fully powered trial of LOCI that is currently published [48], researchers studying mental health care in Norway showed that LOCI improved first-level leaders' implementation leadership, transformational leadership, and clinic implementation climate for trauma-focused EBPs.However, no studies have tested the two key linkages in LOCI's hypothesized theory of change, namely: (1) that improvement in clinic implementation leadership and transformational leadership contributes to subsequently improved clinic implementation climate, and (2) that improvement in clinic implementation climate explains LOCI's effect on EBP fidelity.training and technical assistance only) is expected to cause improvement in clinic-level implementation leadership, transformational leadership, and implementation climate for digital MBC (Aim 1).Improvement in implementation leadership and transformational leadership is expected to mediate LOCI's effect on improved clinic implementation climate (Aim 2).Improvement in clinic implementation climate is expected to mediate LOCI's effect on improved fidelity to digital MBC as experienced by youth (Aim 3).In this study, the clinic level is synonymous with the organization level; however, this is not always the case in applications of LOCI.The LOCI strategy can be applied to organizations with multiple levels, resulting in theoretical models that describe how LOCI intervenes at multiple organizational levels to influence climate

Study contributions
This study makes three contributions to the literature.In Aim 1, we test LOCI's effects on growth in first-level leaders' use of implementation leadership (Hypothesis 1) and transformational leadership (Hypothesis 2), and on clinic implementation climate for MBC (Hypothesis 3), from baseline to 18-month post-baseline (i.e., 6 months after completion of LOCI).This aim seeks to replicate findings from an earlier trial [48] in which LOCI improved these outcomes in a different treatment setting (mental health clinics within Norwegian health trusts), patient population (adults), and set of EBPs (trauma-focused assessment and psychotherapies).In Aim 2, we test the hypotheses that experimentally induced improvement in first-level leaders' implementation leadership (Hypothesis 4) and transformational leadership (Hypothesis 5) at T 2 (4 months after baseline) will mediate LOCI's effects on clinic implementation climate at T 4 (12 months after baseline).In Aim 3, we test LOCI's focal mechanism, namely: that improvement in clinic implementation climate from pre-to post-LOCI (i.e., T 1 to T 4 ) will mediate LOCI's effect on MBC fidelity during the same period (Hypothesis 6).We believe this is the first study to test whether experimentally induced improvement in clinic leadership contributes to improved implementation climate and whether improvement in implementation climate improves observed fidelity to an EBP.

Study design and procedure
Project WISDOM was a cluster randomized, controlled, hybrid type III effectiveness-implementation trial designed to test the effects of LOCI versus training and technical assistance only on MBC fidelity in outpatient mental health clinics serving youth.Details of the trial and primary implementation and clinical outcomes are reported elsewhere [12].The trial enrolled 21 clinics serving youth in Idaho, Oregon, and Nevada, USA.Clinics were eligible if they were not actively implementing a digital MBC system and if they employed three or more clinicians delivering psychotherapy to youth (ages 4-18 years).Using covariate constrained randomization, clinics were randomly assigned to one of two parallel arms: (1) LOCI plus training and technical assistance in MBC or (2) training and technical assistance in MBC only.Clinic-level randomization aligned with the scope of the LOCI strategy and prevented contamination of outcomes at the clinician and patient levels.Clinic leaders could not be naïve to condition; however, clinicians and caregivers of youth were naïve to condition.
Following baseline assessments and randomization of clinics, executives and first-level leaders in the LOCI condition began participating in the LOCI implementation strategy.One month later, clinicians who worked with youths in both conditions received training to implement an evidence-based digital MBC system called the Outcome Questionnaire-Analyst (OQ-A; see below for details; [49,50]).Following the initial OQ-A training, clinics in both conditions received two booster trainings and ongoing OQ-A technical assistance from the OQ-A purveyor organization until the trial's conclusion.
To assess LOCI's effects on its targeted mechanisms of change, clinicians who served youth in participating clinics were asked to complete web-based assessments evaluating their clinic's leadership and clinic implementation climate for MBC at five time points: baseline (T 1 ; following randomization of clinics but prior to initiation of LOCI or OQ-A training), 4-month post-baseline (T 2 ), 8-month post-baseline (T 3 ), 12-month post-baseline (T 4 ; coinciding with the conclusion of LOCI), and 18-month post-baseline (T 5 ; 6 months after LOCI concluded).Surveys were administered from October 2019 to May 2021.Clinic leaders provided the research team with rosters and emails of all youth-serving clinicians at each time point.Confidential survey links were distributed by the research team directly to clinicians via email.Clinicians received a small financial incentive for completion of each assessment (i.e., gift card to a national retailer) based on an escalating structure (US $30, US $30, US $45, US $50, US $55).
The primary implementation outcome of MBC fidelity was assessed for new youth outpatients who initiated treatment in the 12 months following clinician training in the MBC system.Upon intake to services, parents/ caregivers of new youth patients were presented with study information requesting their consent for contact by the research team.Caregivers who agreed were contacted by research staff via telephone to complete screening, informed consent, and baseline measures (if eligible).After study entry, caregivers completed assessments reporting on the youth's treatment participation (i.e., number of sessions) and symptoms monthly for 6 months following the youth's baseline.Assessments were completed regardless of the youth's continued participation in treatment, unless the caregiver formally withdrew (n = 7).Caregivers received a US $15 gift card to a national retailer for completion of each assessment.Enrollment and collection of follow-up data for youth occurred from January 2020 to July 2021.The CONSORT and Stari guidelines were used to report the results of this mediation analysis within the larger trial [51,52].

Participants
All licensed clinicians who worked with youth in participating clinics at each time point were eligible to participate in web-based surveys of clinic leadership and climate.This broad inclusion criterion ensured a full picture of clinic leadership and climate at each time point.
Inclusion criteria for youth were intentionally broad to reflect the trial's pragmatic nature and the applicability of MBC to a wide range of mental health diagnoses.Eligible youth were new patients (i.e., no psychotherapy at the clinic in the prior 12 months), ages 4 to 17 years, who had been diagnosed by clinic staff with an Axis I DSM disorder deemed appropriate for outpatient treatment at the clinic; it was not required that youths be assigned to clinicians who completed surveys.Youths were excluded if they initiated treatment more than 7 days before the informed consent interview.Electronic informed consent was obtained from all participants.The Boise State University Institutional Review Board provided oversight for the trial (protocol no.041-SB19-081) which was prospectively registered at ClinicalTrials.gov(identifier: NCT04096274).

Clinical intervention: digital measurement-based care
The OQ-A is a digital MBC system shown to improve the effectiveness of mental health services in over a dozen clinical trials across four countries [49,53].OQ-A measures are sensitive to change upon weekly administration and designed to detect treatment progress regardless of treatment protocol, patient diagnosis, or clinician discipline [53].In this study, clinicians had access to parent-and youth-report forms of the Youth Outcomes Questionnaire 30.2 [54,55] and the Treatment Support Measure [56,57].Measures were completed by caregivers and/or youth electronically (via tablet or phone).Administration typically took 3-5 min.Measures were automatically scored by the OQ-A system, and feedback reports were generated within seconds.Feedback included a graph of change in the youth's symptoms, critical items (e.g., feelings of aggression), and a color-coded alert, generated by an empirical algorithm, indicating whether the youth was making expected progress or was at risk of negative treatment outcome.
Clinicians were instructed to administer a youth symptom measure to the caregiver and/or youth at each session, review the feedback within 7 days of the session, and use the feedback to guide clinical decision-making.Clinicians were encouraged to discuss feedback with the caregiver and/or youth when they believed it was clinically appropriate and to administer a Treatment Support Measure if a youth was identified as high risk for negative outcome.Consistent with prior MBC trials, clinicians were not given specific guidance on how to respond to feedback; instead, they were advised to use their clinical skills in partnership with patients and clinical supervisors.

Implementation strategies OQ-A training and technical assistance
The initial, 6-h, OQ-A training provided to clinicians in both conditions was conducted in-person by the OQ-A purveyor organization.Training focused on the conceptual and psychometric foundations of the measures, the value of clinical feedback, clinical application of measures and feedback with youth and families, and technical usage of the system.Learning activities included didactics, in vivo modeling and behavioral rehearsal, exercises with sample feedback reports, and use of the system in "playground mode." Two, live, virtual, 1-h booster trainings were offered to clinicians 3 and 5 months after the initial training.Professional continuing education hours were offered at no cost for all trainings to encourage participation.After the initial training, all clinics in both conditions received year-round technical assistance from the OQ-A purveyor organization.This included on-demand virtual training sessions, an online library of training videos, and a customer care representative to troubleshoot technical issues.

Leadership and Organizational Change for Implementation (LOCI)
Details of the LOCI implementation strategy are available elsewhere [11,12].Briefly, LOCI was implemented in quarterly cycles over 12 months.During each cycle, (1) executives and first-level leaders within LOCI clinics attended monthly organizational strategy meetings to review data and to develop clinic-wide policies, procedures, and practices to support OQ-A implementation, and (2) first-level leaders attended leadership development trainings (5 days total) and participated in brief (~ 15 min) weekly coaching calls, designed to enhance their leadership skills.Once per month, individual coaching calls were replaced by group coaching calls with all other first-level leaders in the LOCI condition.
To support enrollment in the study, clinic leaders in the training and technical assistance only condition were offered access to four, professionally produced, webbased, general leadership seminars (1 h each).Seminars covered general leadership topics like giving effective feedback and leading change.The seminars were made available immediately after the OQ-A training.

MBC fidelity
The primary outcome of MBC fidelity was measured at the youth level using an empirically validated MBC fidelity index [22,58,59].The index was generated using electronic metadata from the OQ-A system combined with monthly caregiver reports of the number of sessions youths attended.Following prior research [22], scores were calculated as the product of two quantities: (a) the youth's completion rate (i.e., number of measures administered relative to the number of sessions attended within the 6-month observation period) and (b) the youth's viewing rate (i.e., the number of feedback reports viewed by the clinician relative to the number of measures administered).Note that this product is equivalent to the ratio of viewed feedback reports to total sessions; it represents an events/trials proportion.MBC fidelity index scores summarize the level of MBC fidelity experienced by each youth (range = 0-1) and have been shown to predict clinical improvement of youths receiving MBC [22,58,59].Importantly, this index captures the administration and viewing components of MBC fidelity but does not indicate whether clinicians used the feedback to guide treatment.

Implementation leadership
Clinicians assessed the extent to which their first-level leaders exhibited implementation leadership behaviors with regard to the OQ-A using the 12-item Implementation Leadership Scale (ILS) [40].The ILS includes four subscales assessing the extent to which the firstlevel leader is proactive, knowledgeable, supportive, and perseverant about implementation.Responses were made on a 0 (not at all) to 4 (very great extent) scale.Total scores were calculated as the mean of all items.In prior research, scores on the ILS exhibited excellent internal consistency, convergent and discriminant validity [40,47,60,61], and sensitivity to change [45].

Transformational leadership
Clinicians assessed the extent to which their first-level leaders exhibited transformational leadership behaviors using the Multifactor Leadership Questionnaire (MLQ) [62,63].The MLQ is a widely used measure that has demonstrated excellent psychometric properties [64] and is associated with implementation climate for EBP as well as clinicians' attitudes toward, and use of, EBPs [65][66][67][68].
Responses were made on a 5-point scale ("not at all" to "frequently, if not always").Consistent with prior studies, we used the 20-item transformational leadership total score, calculated as the mean of four subscales: idealized influence, inspirational motivation, intellectual stimulation, and individual consideration.

Clinic implementation climate
Clinicians' perceptions of their clinics' implementation climate for OQ-A were measured using the 18-item Implementation Climate Scale (ICS) [34].The ICS includes six subscales assessing focus, educational support, recognition, rewards, selection, and openness.
Responses were made on a 0 (not at all) to 4 (a very great extent) scale with the total score calculated as the mean of all items.Prior research provides evidence for the structural, convergent, and discriminant validity of scores on the ICS [27,34,[69][70][71][72] as well as sensitivity to change [45].

Data aggregation
Best practice guidelines [73][74][75][76] recommend clinician ratings of first-level leadership and clinic implementation climate be aggregated and analyzed at the clinic level.To justify aggregation, guidelines recommend that researchers test the level of inter-rater agreement among clinicians within each clinic to confirm there is evidence of shared experience.We used the r wg(j) statistic [77] to assess interrater agreement among clinicians within each clinic.Across all clinics and all waves, average values of r wg(j) were above the recommended cutoff of 0.7 [78,79] for implementation leadership (M = 0.82, SD = 0.27), transformational leadership (M = 0.87, SD = 0.24), and clinic implementation climate (M = 0.94, SD = 0.10).

Covariates
In order to increase statistical power and to address potential imbalance across clusters, we planned a priori to include covariates of state and clinic size (number of youths served in the prior year) in all analyses.In addition, in the mediation analyses described below, we included baseline values of the hypothesized mediator and outcome (when possible) to increase the plausibility of the no-unmeasured-confounding assumptions within the causal mediation approach [80,81].

Data analysis
All analyses used an intent-to-treat approach.To test LOCI's effects on growth in first-level leaders' implementation leadership (H1), transformational leadership (H2), and clinic implementation climate (H3) for Aim 1, we used three-level linear mixed-effects regression models [82,83] with random effects addressing the nesting of repeated observations (level 1) within clinicians (level 2) within clinics (level 3).Separate models were estimated for each outcome.At level 1, observations of leadership and climate collected from clinicians at each time point were modeled using a piecewise growth function that captured differences in change from baseline to each time point across conditions [84].Implementation condition and clinic covariates were entered at level 3. Models were estimated using the mixed command in Stata 17.0 [85] under full maximum likelihood estimation, which accounts for missing data on the outcomes, assuming data are missing at random.Effect sizes were calculated as the standardized mean difference in change (i.e., difference in differences) from baseline to each time point (i.e., Cohen's d) using formulas by Feingold [86].Cohen suggested values of d could be interpreted as small (0.2), medium (0.5), and large (0.8) [87].
Aim 2 tested the hypotheses that experimentally induced improvement in first-level leaders' implementation leadership (H4), and transformational leadership (H5) by T 2 , would mediate LOCI's effect on improvement in clinic implementation climate by T 4 .These mediation hypotheses were tested using the multilevel causal mediation approach by Imai et al. [81], implemented in the R "mediation" package [88].To align our analytic approach with our theoretical model, we estimated a 2-2-1 mediation model in which the primary antecedent (LOCI) and mediator (clinic-level aggregate leadership scores) entered the model at level 2 (i.e., the clinic level), and the outcome (clinician ratings of implementation climate) entered at level 1, representing latent clinic means [89].Separate models were estimated for each type of leadership because Imai's approach does not accommodate simultaneous mediators [81].The inclusion of baseline values for the mediator (i.e., leadership) and outcome (i.e., climate) in each model modified the interpretation of the effects so that they represented the effect of LOCI on change in leadership from T 1 to T 2 and of change in leadership on change in climate from T 1 to T 4 .To stabilize the effect estimates, we set the number of analytic simulations for the direct and indirect effects to 10,000.These analyses produced estimates of LOCI's indirect and direct effects on T 4 implementation climate, as well as the proportion of LOCI's total effect on implementation climate that was mediated by improvement in leadership (i.e., proportion mediated = p m ).Indirect effects indicate the extent to which LOCI influenced T 4 implementation climate through its effect on T 2 leadership (i.e., mediation).Direct effects indicate the residual (remaining) effect of LOCI on T 4 implementation climate that was not explained by change in T 2 leadership.The p m statistic is an effect size measure indicating how much of LOCI's effect on implementation climate was explained by change in leadership.
Aim 3 tested the hypothesis that improvement in clinic implementation climate from T 1 to T 4 would mediate LOCI's effect on MBC fidelity during the same time period (H6).The nested data structure was accommodated using a 2-2-1 model in which the primary antecedent (LOCI) and mediator (aggregate clinic-level T 4 implementation climate scores) occurred at level 2 (i.e., clinic level) and the outcome (MBC fidelity) occurred at level 1 (i.e., youth level).Note that the inclusion of baseline values of clinic implementation climate in this model modified the interpretation of the effects so that they represent the effect of LOCI on change in climate from T 1 to T 4 and of change in climate on fidelity during the same time period.To address the events/trials nature of the MBC fidelity index, a generalized linear mixedeffects model with random clinic intercepts, a binomial response distribution, and a logit link function was used in the second step of the mediation analysis [82].In total, 18 clinics enrolled a total of 234 youth, all of whom had MBC fidelity data; however, one clinic was missing ratings of T 4 implementation climate, resulting in a sample of 17 clinics and 231 youth for this analysis.A sensitivity analysis based on mean imputation of the missing T 4 implementation climate value yielded the same inferential conclusions.A priori statistical power analyses conducted with the PowerUp!macro [90,91] indicated the trial had power of 0.74-0.90 to detect minimally meaningful mediation effect sizes depending on observed intraclass correlation coefficients and variance explained by covariates.

Results
Figure 2 shows the flow of clinics, clinicians, and youth through the study.As is shown in Table 1, there were no differences by condition on the distribution of any clinic (K = 21), clinician (N = 252), or youth (N = 231) characteristics (all ps > 0.05).In total, 252 clinicians completed assessments for the study across 5 waves (average response rate = 88% across waves).The average number of participating clinicians per clinic was 12 (SD = 6.4).Nearly two-thirds of clinicians (n = 154, 61%) participated in 3 or more waves of data collection, and there were no differences by condition on clinician participation patterns (p = 0.114).A total of 234 youths were enrolled in the study.The average number of youths per clinic was 13.6 (SD = 13.3).The average number of assessments completed per youth was 5.9 (SD = 1.8) out of 7; 64% (n = 148) of youth had complete data, and 90% (n = 208) had 3 or more completed assessments.There were no differences in caregiver response rates for youth data by condition (p = 0.557).

Indirect effects of LOCI on T 4 implementation climate through T 2 clinic leadership
Hypotheses 4 and 5 examined how LOCI improved T 4 clinic implementation climate by testing mediation models.Hypothesis 4 stated LOCI would have an indirect effect on T 4 implementation climate through improvement in first-level leaders' use of implementation leadership from T 1 to T 2 .As is shown in Table 3, this hypothesis was supported.The LOCI strategy had To accommodate the nesting of clinicians and youths within clinics as well as multiple testing, p-values were generated using a bootstrap alpha on the basis of 500 resamples with replacement.GED, general education degree; LOCI, Leadership and Organizational Change for Implementation strategy; M, mean; SAC, Shortform Assessment for Children; SD, standard deviation  a significant indirect effect on T 4 implementation climate through T 2 implementation leadership (indirect effect = 0.51, p = 0.004) even as LOCI's direct effect was not statistically significant (direct effect = 0.11, p = 0.582).
The proportion-mediated statistic indicated 82% of LOCI's total effect on clinic implementation climate at T 4 was explained by improvement in implementation leadership from T 1 to T 2 (p m = 0.82).Hypothesis 5 stated LOCI would have an indirect effect on T 4 implementation climate through improvement in first-level leaders' transformational leadership.This hypothesis was not supported (see Table 3).There was no evidence of an indirect effect of LOCI through transformational leadership (indirect effect = 0.16, p = 0.135) even as LOCI's direct effect on T 4 implementation climate remained statistically significant (direct effect = 0.38, p = 0.024).This pattern confirms LOCI improved T 4 implementation climate but not through its effect on transformational leadership.

Indirect effect of LOCI on MBC fidelity through implementation climate
Hypothesis 6 stated that LOCI's effect on clinic implementation climate from T 1 to T 4 would mediate LOCI's effect on MBC fidelity measured at the youth level during the same time period.As is shown in Table 3, results of the mediation analysis supported this hypothesis.The LOCI strategy had a statistically significant indirect effect on MBC fidelity through clinic implementation climate, increasing fidelity by 14 percentage points (indirect effect = 0.14, 95% CI = 0.01-0.37,p = 0.033) through this mechanism.The direct effect of LOCI on fidelity after accounting for clinic implementation climate was not statistically significant (direct effect = 0.05, p = 0.482).The proportion-mediated statistic indicated 71% of LOCI's effect on MBC fidelity was explained by improvement in clinic implementation climate from T 1 to T 4 (p m = 0.71, p = 0.045).In this trial, increases in first-level leaders' use of implementation leadership by 4-month post-baseline explained 82% of LOCI's effect on improvement in clinic implementation climate by 12-month post-baseline.This finding aligns with qualitative data from another recent trial of MBC implementation in community mental health [93], which also found that leader and clinical supervisor support for MBC were perceived as key implementation mechanisms.The linkage of implementation leadership to improvement in clinic implementation climate suggests implementation leadership behaviors are important targets for implementation success.Accordingly, pre-service educational programs for health leaders, implementation purveyor organizations, and other stakeholders interested in supporting EBP implementation should consider integrating these leadership competencies into core curricula and training.
Contrary to our hypotheses (see Fig. 1), LOCI did not exert lasting effects on transformational leadership, and increases in transformational leadership did not explain LOCI's effect on improvement in clinic implementation climate.This pattern of results aligns with theoretical models of leadership and climate which suggest that specific types of focused leadership (i.e., implementation leadership [40]) are needed to generate specific types of focused organizational climate and associated outcomes [37,44].This finding also suggests the LOCI strategy could be streamlined without loss of efficacy by scaling back (or eliminating) components that address transformational leadership, as has been done in implementation studies of autism interventions [94].Streamlining the content of LOCI may increase LOCI's feasibility and allow for greater development of implementation leadership skills.
Improvement in clinic implementation climate explained 71% of LOCI's effect on youth-level MBC fidelity.The validation of this theoretical linkage within an experimental design lends credence to prior correlational studies and theory suggesting clinic implementation climate can contribute to improved EBP implementation in mental health settings [27,[44][45][46][47].These results suggest organizational and system leaders can improve the implementation of EBPs by deploying organizational policies, procedures, and practices that send clear signals to clinicians about the importance of EBP implementation relative to competing priorities within practice settings.
Discussions about change in organizational leadership and organizational climate often center around how long it takes for these constructs to change and what level of resources are required.Results from this trial suggest changes in first-level leadership and clinic implementation climate can occur quickly, within 4 months, and that these changes can be lasting, even 6 months after supports (i.e., LOCI) are removed.A similarly brief timeframe for initial change, and similarly sustained period of maintenance of effect, was observed for implementation leadership and climate in the other large trial of LOCI, which occurred in a different country, patient population, and EBP [48].Together, results from these trials confirm implementation leadership and implementation climate are modifiable with a combination of training, weekly coaching calls, data feedback, and goal setting.
This study highlights multiple directions for future research.Future studies should examine moderators of LOCI's effectiveness with an eye toward the minimally necessary components to make LOCI effective.For example, there is some evidence that intervention complexity moderates the association between implementation climate and fidelity [47].This is consistent with organizational climate theory, which indicates climate is most strongly related to employee behaviors when service complexity is higher [95] and when there is high interdependence among employees to complete tasks and high intangibility of the service provided [96].Other research should look beyond the organization level at how systems can be modified in complementary ways to create supportive implementation climates for targeted interventions.
Strengths of this study include the use of an experimental, longitudinal design, enrollment of clinics in diverse policy environments (i.e., three different States), measurement of leadership and climate by third-party informants whose behavior is most salient to implementation success (i.e., clinicians), measurement of MBC fidelity through objective computer-generated data, time ordering of hypothesized antecedents and consequents, and use of rigorous causal mediation models to estimate direct and indirect effects.The study's primary limitation is generalizability given all clinics and caregivers of youth volunteered to participate.In addition, the MBC fidelity measure does not assess whether clinicians used the feedback to inform clinical decisions.Data were not collected on other mechanisms that may explain LOCI's effects, a gap that may be fruitfully addressed by future qualitative research.Because the LOCI condition included training and technical assistance, it was not possible to isolate LOCI's independent effects; this is also a fruitful area for future research.Finally, we were unable to fully test LOCI's hypothesized theory of change due to our use of the causal mediation approach which precludes testing serial multiple mediator models (e.g., LOCI → leadership → climate → fidelity).Nonetheless, our results confirm the most consequential links in LOCI's theoretical model and offer important directions for research and practice.

Conclusion
In this mediation analysis of the WISDOM trial, experimentally induced improvement in implementation leadership explained increases in clinic implementation climate, which in turn explained LOCI's effects on MBC fidelity in youth mental health services.This offers strong evidence that fidelity to EBPs can be improved by developing organizational leaders and strong implementation climates.

Fig. 1
Fig. 1 Study theoretical model.Note: LOCI, Leadership and Organizational Change for Implementation strategy; MBC, measurement-based care.First-level leaders are those who administratively supervise clinicians (e.g., clinical managers).Random assignment of clinics to LOCI (vs.training and technical assistance only) is expected to cause improvement in clinic-level implementation leadership, transformational leadership, and implementation climate for digital MBC (Aim 1).Improvement in implementation leadership and transformational leadership is expected to mediate LOCI's effect on improved clinic implementation climate (Aim 2).Improvement in clinic implementation climate is expected to mediate LOCI's effect on improved fidelity to digital MBC as experienced by youth (Aim 3).In this study, the clinic level is synonymous with the organization level; however, this is not always the case in applications of LOCI.The LOCI strategy can be applied to organizations with multiple levels, resulting in theoretical models that describe how LOCI intervenes at multiple organizational levels to influence climate

Fig. 2
Fig. 2 CONSORT diagram showing the flow of clinics, clinicians, and youth through the WISDOM trial Note: ITT, intent to treat; LOCI, Leadership and Organizational Change for Implementation strategy; T 2 , 4-month follow-up; T 4 , 12-month follow-up; WISDOM, Working to Implement and Sustain Digital Outcome Measures trial.a One clinic participated in LOCI for only 6 months.b One clinic that enrolled youth did not have T 4 climate data

Note:
Reliability coefficients calculated as Cronbach's alpha.Cohen d effect sizes express the standardized mean difference in change from baseline to the referenced time point, contrasting clinics assigned to LOCI versus control (training and technical assistance only).Effect sizes generated based on linear mixed-effect models and formulas by Feingold [86]; bolded values are statistically significant at p < .05.CI confidence interval a K = 21 clinics, N = 248 clinicians, J = 803 observations b K = 21 clinics, N = 251 clinicians, and J = 810 observations c K = 21 clinics, N = 247 clinicians, and J = 809 observations

Table 1
Characteristics of participating clinics, clinicians, and youth by condition

Table 2
Effects of the Leadership and Organizational Change for Implementation (LOCI) strategy versus control on clinic leadership and clinic implementation climate by time

Table 3
[88]ct and indirect effects of the Leadership and Organizational Change for Implementation (LOCI) strategy on clinic implementation climate and digital MBC fidelityEffect estimates generated using Imai's multilevel causal mediation R package[88].Estimates in bold are statistically significant at p < .05(two-tailed).All models include implementation condition (random assignment to LOCI vs. training and technical assistance only) as the focal antecedent plus covariates of state, clinic size, and baseline value of the mediator.CI, confidence interval; MBC, measurement-based care; LL, lower limit; T 2 , 4-month post-baseline; T 4 , 12-month post-baseline; UL, upper limit a K = 20 clinics, N = 166 clinicians.These models also include baseline clinic implementation climate as a covariate.Indirect and direct effects indicate the change in T 4 implementation climate (range = 0-4) caused by LOCI b K = 17 clinics, N = 231 youth.Indirect and direct effects indicate the proportion difference in MBC fidelity (range = 0 − 1) caused by LOCILOCI effectOutcome: T